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Abstract.  We  propose  that  a  set  of  five  dimensions  forms  a  foundation  underlying  a  number  of  prevalent 
theoretical  perspectives  on  learning.  We  show  how  student  contributions  to  instructional  dialogue  can  be 
reliably  annotated  with  these  dimensions.  Finally,  we  provide  preliminary  validation  evidence  for  our 
coding  scheme  and  illustrate  the  potential  value  of  such  an  approach  to  analyzing  student  behavior  in 
tutorial  dialogue. 
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Introduction 

It  is  difficult  to  compare  results  across  learning  studies,  because  they  are  often  designed 
to  investigate  different  learning  theories  and  use  difficult  coding  strategies.  To  alleviate 
this  problem  we  want  establish  a  "common  language"  for  coding  tutorial  dialogues. 

First,  we  identified  five  prominent  learning  theories  and  decomposed  them  into  a  set 
of  underlying  dimensions  (see  Table  1).  Next,  we  transformed  those  dimensions  (depth, 
motivation,  accuracy,  novelty  and  doubt  into  a  coding  system,  which  we  labelled 
DeMAND  (see  Table  2). 


Table  1 :  Theories  and  Underlying  Dimensions. 

The  Theories:  Learning  results  from. . .  Can  be  Decomposed  into  these  Underlying  Dimensions 

Applying  cognitive  effort  [  1  ]  Statements  that  are  DEEP 

Engaging  in  accountable  talk  [2]  Statements  that  are  DEEP  and  ACCURATE 

Constructing  new  knowledge  [3,4]  Statements  with  NOVEL  information 

Autonomous  activity  [5]  Statements  that  are  internally  MOTIVATED 

Experiencing  an  impasse  [6]  Statements  that  are  not  ACCURATE  or  show  DOUBT 


In  the  rest  of  this  paper,  we  show  that  this  system  can  be  reliably  applied  to  a  corpus, 
provide  preliminary  evidence  of  its  validity  and  illustrate  its  potential  utility. 
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Table  2:  DeMAND  Codes. 


Code _ 

De(pth) 

M(otivation) 


A(ccuracy) 

N(ovelty) 

D(oubt) 


Definition _ 

Credited  if:  contained  explicit  signs  of  deep  processing,  such  as  explaining,  justifying, 
providing  evidence,  asking  questions  to  gain  a  better  understanding,  etc. 

All  statements  were  coded.  A  statement  was  externally  motivated  if  it  was  made  in  reaction  to 
a  “demand”  of  the  lesson  or  tutor.  A  statement  was  internally  motivated  if  it  could  not  be 
traced  to  any  external  demand. 

All  statements  were  coded.  Possible  values  include:  Correct  &  complete,  correct  but 
incomplete,  partially  correct  with  some  errors  and  incorrect. 

Credited  if:  contained  information  that  has  not  already  been  explicitly  stated  (or  confirmed)  by 
the  tutor,  slides,  or  student. 

Credited  if:  contained  explicit  indicators  of  a  lack  of  confidence,  such  as  hedges. _ 


2.  Current  Study 

We  created  a  curriculum  and  a  computer-based  learning  environment  to  teach  basic 
concepts  in  direct  current  circuits.  The  curriculum  took  students  approximately  4  hours 
to  complete.  The  learning  environment  contained  lesson  materials  (including  didactic 
text,  exercises  and  discussion  questions),  a  circuit  simulator,  and  a  message  window 
where  the  participant  and  tutor  interacted.  All  of  the  student-tutor  interactions  occurred 
through  typing. 

We  created  three  versions  of  these  lessons  (open-ended,  closed-ended,  and  middle), 
which  varied  only  in  the  format  of  the  questions  that  the  students  were  asked.  Our 
objective  was  to  impact  the  type  of  language  produced  by  the  participants,  with  the 
open-ended  condition  eliciting  more  deep  statements,  more  accountable  talk  and  more 
internally  motivated  statements  than  the  closed-ended  condition.  If  our  system  reflects 
these  differences,  this  provides  some  positive  evidence  for  its  construct  validity  [7]. 

The  corpus  includes  dialogues  from  each  of  thirty  participants  distributed  across 
three  experienced  tutors  and  conditions.  The  entire  corpus  includes  8,085  dialogue 
turns  taken  by  the  student  and  tutor,  and  56,133  tokens  (words  and  punctuation). 


3.  Results 

Following  [8],  our  raters  achieved  kappa  values  of  0.64  (“moderate”),  0.81,  0.89  and 
0.88  (“substantial”)  for  depth,  motivation,  novelty  and  doubt,  respectively.  Accuracy 
was  coded  in  real  time  by  the  tutors. 

We  conducted  three  one-way,  between-subjects  analyses  of  variance  (ANOVAs)  with 
lesson  condition  (open,  middle  and  closed)  as  the  independent  variable  for  each,  and 
the  percentage  of  student  utterances  that  were  coded  as  containing  evidence  of  (a) 
cognitive  effort,  (b)  accountable  talk  or  (c)  internally  motivated  as  the  three  dependent 
variables  (DVs),  respectively.  In  each  case,  the  overall  ANOVA  was  significant,  and  a 
Fischer’s  least  significant  difference  (LSD)  test  indicated  that  the  DV  was  larger  in  the 
open  condition  than  in  the  closed  condition  (see  Table  3). 

We  conducted  a  hierarchical  multiple  linear  regression,  with  post-test  score  as  the 
dependent  variable  and  the  pre-test  score  and  the  five  learning  theories  (cognitive 
effort,  internal  motivation,  constructivism,  accountable  talk,  and  experiencing 
impasses)  as  independent  variables.  The  result  was  significant,  F(l,28)  =  8.1,  p~008, 


with  R 2  =  0.224.  The  only  significant  predictor  was  impasses,  which  were  negatively 
related  to  post-test  score,  /?=-0.47,  t=-2.85,/>=.008. 


Table  3:  Statistics. 


ANOVA  LSD  Descriptive  Statistics  Descriptive  Statistics  for 

_ for  Open  Condition _ Closed  Condition 

(a)  DV  =  percentage  of  student  utterances  coded  as  containing  evidence  of  cognitive  effort 

F( 2,29)  =  66.13, p<.0\  t(  19)  =  10.67, p<.0\  M=  .45,  SD  =  .07  M=  .13,  SD  =  .07 

(b)  DV  =  percentage  of  student  utterances  coded  as  accountable  talk 

F( 2,29)  =  59.82,  p<.0\  t(  19)  =  9.67,p<.01  M=  .41,  SD  =  .07  M=  .11,  SD  =  .07 

(c)  DV  =  percentage  of  student  utterances  coded  as  internally  motivated 

F( 2,29)  -  6.22,  p  =  .006  t{  19)  =  4.00,  p  =  .003  M=  .07,  SD  =  .04 _ M=  .02,  SD  =  .02 


4.  Summary 

Without  a  common  language,  it  can  be  difficult  to  compare  results  across  different 
learning  studies.  We  proposed  that  five  prominent  theories  of  learning  could  be 
represented  by  a  series  of  underlying  dimensions  (some  singly,  others  with 
combinations).  We  showed  that  these  dimensions  could  be  reliably  assessed  and  that 
they  respond  as  predicted  to  an  instructional  manipulation  (an  accepted  approach  for 
collecting  evidence  of  construct  validity  [7]).  Finally,  we  showed  that  this  coding 
scheme  allowed  us  to  simultaneously  evaluate  the  relative  explanatory  power  of  the 
different  learning  theories  for  our  students’  learning  gains. 

While  additional  work  is  required  to  refine  and  evaluate  this  approach,  we  believe 
that  decomposing  learning  theories  into  a  set  of  “least  common  denominators”  as  we 
have  done  here  will  facilitate  a  deeper  understanding  of  the  relationships  between 
theories,  allow  us  to  more  readily  compare  results  across  studies  and  support  a 
systematic  investigation  into  the  relative  impact  on  learning  gains  of  different  types  of 
events,  experiences  and  pedagogical  methods. 
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